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The four papers presented in this symposium report on the evaluation strategies and 
feedback from teachers as they embarked on designing and implementing STEM 
approaches to learning in secondary school contexts. The STEM professional learning 
program was designed to provide time and expert support so that cross-disciplinary school 
teams of up to six teachers from science, mathematics and technology/engineering could 
develop new school-based initiatives. A range of evaluation strategies were used in the first 
two STEM Academy programs to identify factors and approaches that supported teachers’ 
and students’ needs, and to further enhance the STEM Academy program. This symposium 
addresses ways in which the progressive evaluations have informed this change process. 


Paper 1: Judy Anderson, The University of Sydney. The STEM Teacher Enrichment 
Academy Approach 
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The STEM Teacher Enrichment Academy Approach 
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The STEM Teacher Enrichment Academy was developed to promote the STEM subjects in 
schools so that more students engage with, and consider pursuing a STEM-based career. 
Since teachers are key to student engagement and interest, a professional learning program 
for multi-disciplinary school teams was developed to support teachers in identifying and 
designing the most appropriate STEM approach for their students. Offered for the first time 
in 2014, the Academy has been implemented four times, reaching 260 secondary teachers 
from 47 schools. Feedback using a range of data collection tools has enabled the evolution 
of the program to better address school, teacher and student needs. 


With a global decline in students enrolling in mathematics and science subjects at the 
senior secondary and tertiary levels (Kennedy, Lyons, & Quinn, 2014), and predictions that 
we will need many more mathematicians and scientists to meet workplace demands of 
STEM (Science, Technology, Engineering, and Mathematics) related professionals into the 
future (Office of the Chief Scientist, 2016), school systems and other stakeholders have 
embarked on developing new approaches to promoting STEM. To build on, and coordinate 
the range of reforms, a STEM Education Forum was held in Sydney in 2015 to develop a 
National STEM School Education Strategy, 2016-2026 (National Council, 2015). Driven 
by the two key goals of wanting all students to finish school with strong foundational 
STEM skills and capabilities, and ensuring all students want to embark on more 
challenging STEM subjects, the Strategy identified five key areas for national action: 

1. increasing student STEM ability, engagement, participation and aspiration; 

2. increasing teacher capacity and STEM teaching quality; 

3. supporting STEM education opportunities within school systems; 

4. facilitating effective partnerships with tertiary education providers, business 

and industry; and 

5. building a strong evidence base. 
It is noted in the Strategy that these key actions relate to both the individual STEM subjects 
as well as to any integrated approaches to STEM education. 

While many factors influence student participation as measured through subject choice 
and subject engagement in secondary schooling, McPhan, Morony, Pegg, Cooksey and 
Lynch (2008) determined the lower participation of students in senior mathematics was 
particularly influenced by poor pedagogical practices, perceived level of difficulty, and 
irrelevance. Traditional approaches to teaching mathematics and science do not capture the 
multi-disciplinary nature of contemporary mathematics and science practices (Tytler, 
Symington, & Smith, 2011) and their connections to the systems thinking, design thinking, 
or computational thinking of engineering and technology (Bybee, 2013; English, 2016). 
Proponents of integrated STEM curriculum argue for its potential to increase student 
motivation and engagement (Beane, 1993); to enable students to transfer knowledge, make 
connections and see the relevance of the STEM subjects (English, 2016); to develop 
students’ “STEM literacy” and their understanding of local and global challenges (Bybee, 
2013); and to provide an impetus towards the further study of STEM subjects in senior 
schooling and STEM degrees at university (Freeman, Marginson, & Tytler, 2015). 
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To date, there has been little research conducted into the efficacy of STEM subject 
integration in secondary classrooms (Bruder & Prescott, 2013), but there is some evidence 
to suggest that STEM integration is successful in increasing student engagement within 
mathematics classrooms (Venville, Wallace, Rennie, & Malone, 1998). Based on the 
assumption that students benefit from opportunities to connect knowledge across the 
curriculum, a professional learning approach was developed to support teachers in 
planning and implementing connected approaches in secondary schools. The design of the 
Academy program was informed by research into effective professional learning practices 
but with little research available about the best approaches to integrated STEM learning 
more generally, the program has evolved based on feedback from teachers’ experiences. 


The Initial Academy Design 


In 2014, the Faculty of Education and Social Work collaborated with the Faculties of 
Science, and Engineering and Information Technology, to develop the initial program of 
teacher enrichment and professional development. The multi-day on campus program for 
up to 75 teachers (from 12 schools) of Year 7-10 mathematics, science and 
technology/engineering was designed to be foundational in enhancing teachers’ knowledge 
of content and pedagogy, inspiring them to reinvigorate their classroom practice and 
improve student engagement in STEM subjects. The overall Academy aims were to: 

* introduce and support exciting and effective approaches to learning, enhance 
teachers’ knowledge of content and approaches to teaching mathematics, 
science and digital technologies in Years 7-10; 

¢ develop a community of practice for participating STEM teachers, with 
ongoing support and engagement through mentoring, online forums, 
newsletters, seminars and events; and 

¢ develop teachers’ knowledge of STEM-related research and industry as well as 
knowledge of STEM programs at university and career pathways. 

Modelled on commonly agreed core features, the Academy professional learning 
approach was developed to incorporate a content focus, active learning, coherence, 
duration and collective participation (Desimone, 2009). With a focus on examining content 
and processes from the STEM subjects, Academy sessions were facilitated by the 
University’s academic specialists and STEM leaders, as well as teacher/peer-led sessions. 
The program involved a three-day on campus program at the University followed by up to 
two full school terms working on developing, planning and implementing STEM strategies 
in school-based teams. Teachers then returned for a further two-day program at the 
University to share their experiences, present evidence of teacher and student learning, 
discuss issues and challenges, and consider future initiatives. Each cross-disciplinary 
school team of two mathematics, two science and two technology teachers worked together 
to develop inquiry-based learning approaches to teaching both within their subject 
discipline as well as across the subject disciplines (Maa & Artigue, 2013). Initially 
focusing on the individual STEM subjects was adopted because mathematics and science 
teachers made limited use of inquiry-based learning approaches in lessons that is 
recommended in curriculum documents and in research into meaningful learning (Sullivan, 
2011; Tytler, 2007). 


The First and Second Academies 


For the first Academy, 64 teachers from 13 schools visited the University in November 
2014 and returned in March 2015 (see Table 1 for sector representation) — schools were 
invited to participate based on engagement with the University. While most schools are 
Sydney based, four are clustered near Mudgee in the central West of NSW. This small 
country hub of schools enabled greater opportunity for collegiality, an essential ingredient 
given the small size of these schools with some teachers reporting feeling isolated and with 
limited access to quality professional learning. Like the first Academy, the second involved 
70 teachers from 12 schools with a country hub of two larger schools from Wagga Wagga 
(see Table 1) and took place in November 2015 with a subsequent return to the University 
in May 2016. When selecting each group of schools, we sought diversity in school 
systems, socio-economic status, gender composition, and size to further expose teachers to 
the range of issues involved in curriculum redesign and promote community engagement. 


Table 1 
School Sector Representation for the First Two STEM Academies Including School Gender 
Composition 


Department of Education Catholic Independent Total 
2014/15 8 (1 all girls) 1 4 (2 all boys, 2 all girls) 13 
2015/16 7 (1 all boys) 2 (1 all girls) 3 (1 all boys, 1 all girls) 12 


While overall the feedback from teachers has been positive, the key challenges to be 
addressed based on the first two academies included implementing inquiry-based learning 
approaches in regular classrooms, understanding the connections between the separate 
STEM subjects, working effectively in school teams, designing a STEM strategy most 
suitable for school contexts, and building the community of practice. Further detail about 
the evaluations of each of these first two academies is presented in the second and third 
papers in this symposium. 

Our experiences from both academies revealed some schools move more quickly to 
developing integrated STEM approaches because of experiences prior to academy 
participation of writing integrated units of work, and/or working together as a team. This 
highlighted the diversity of teachers’ knowledge and experiences of integrated STEM 
before coming to the Academy and the influence this had on their progress within the 
Academy. Some teams were cohesive while others were dominated by one or two teachers 
who already had a plan which would be implemented regardless, while others had never 
worked together on creative programming and curriculum design. It became clear that we 
needed to conduct school audits of their STEM work as well as to consider teachers’ 
experiences of working together before they arrived to participate in the program. 

Team building and effective whole school planning have now become critical 
components of the Academy and these begin with each school before they attend the first 
session at the University. On site, preliminary planning meetings include the school 
principal and other school leaders who need to play a key role in supporting the 
development of STEM initiatives which frequently have implications for timetabling, 
teacher allocation to classes, alignment of STEM subjects on timetable lines, and 
resourcing. Schools have adopted a wide variety of approaches to implementing STEM 
education — frequently these decisions have been based on available personnel, teacher 
interest and resources but school structures can act as impediments to innovative practices. 


Because the schools were so diverse, particularly in relation to teachers from different 
subjects working together, the approaches they initially adopted were equally disparate. 
From embedding more cross-curriculum applications within regular lessons to conducting 
cross-disciplinary investigations in several STEM subject lessons, schools adapted and 
designed their approaches around perceived student needs sometimes finding lateral ways 
to overcome constraints from school structures and resources. Our purposeful tolerance for 
such diversity acknowledges that schools need to consider the needs of their students, the 
competence and interest of teachers, the overwhelming influence of siloed assessment in 
many schools, and that real change takes time. 

Building the community of practice has been a challenge. While on campus at the 
university, teachers willingly discussed ideas with teachers from other schools, and 
engaged in worthwhile sharing of ideas but the busyness of school life frequently meant 
little ongoing sharing in the online community. In some schools, finding time to meet as a 
school team was enough of a challenge and proved to be an inhibiting factor in moving 
plans forward. To alleviate some of these challenges, for schools to become STEM 
Academy participants, we had requested principals provide time for teachers to work on 
their projects. Unfortunately, this was not always achieved and some academy teachers 
have admitted this is as much they themselves not wanting to take time away from 
something else. Teachers being provided with school time to work on their projects, and 
accepting to do so remains another challenge to be addressed. 
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The initial STEM Teacher Enrichment Academy was held in 2014/2015 involving 64 
teachers from 13 schools. The teachers attended two on-campus sessions which bookended 
their STEM work in schools supported by Academy mentors and online interactions using 
Edmodo. In survey responses, the teachers reported that the Academy had extended their 
pedagogical knowledge for engaging students in STEM and to a lesser extent their STEM 
content knowledge, however they all valued the networking opportunities afforded by the 
Academy. Despite their enthusiasm for implementing new STEM activities in their schools, 
teachers were significantly challenged by a lack of time to plan adequately. 


The initial STEM Teacher Enrichment Academy (2014/2015) was designed to improve 
STEM teachers’ capacity to plan and implement engaging STEM lessons for Years 7 to 10, 
provide opportunities for within and cross-school collaboration and to improve teachers’ 
knowledge of STEM industries and tertiary STEM study options. The Academy pre- 
empted the National STEM School Education Strategy, 2016-2026 (National Council, 
2015) but its goals were closely aligned with the five areas for action identified later in the 
Strategy. In this sense, the Academy has acted as a forerunner in providing leadership and 
direction for STEM teacher development in recent years. 

Currently teachers and school systems are grappling with forecasts for new student 
skill sets requiring integrated models of curriculum development (ITL Research, 2011), 
particularly in the STEM areas where skill shortages are predicted. The National STEM 
School Education Strategy identifies the need for students to develop scientific, 
mathematical and technological literacy along with 21st century skills such as problem 
solving, critical analysis and creative thinking (National Council, 2015). These ‘softer 
skills’ are also embedded in the cross-curriculum capabilities identified in the Australian 
Curriculum (ACARA, 2015). The initial STEM Academy was designed with these 
advances in mind, aiming to prepare teachers to foster inquiry approaches to the teaching 
of STEM subjects so that students’ interest in STEM and development of ‘21st century 
skills’ could flourish. 


The Initial STEM Academy Evaluation Design 


The evaluation of the initial STEM Academy was designed as a mixed-method, survey 
and interview study. All teachers (n = 64) completed surveys during the on-campus 
sessions and a sample of volunteer teachers (n = 22) also participated in telephone 
interviews after the second on-campus session. The three Academy mentors also took part 
in a telephone interview at the end of the Academy. 

The survey instrument was designed to provide a measure of teachers’ existing 
pedagogical practices prior and subsequent to participating in the Academy. The survey 
items were derived from the Innovative Teaching and Learning survey instrument (ITL 
Research, 2011) which provides measures of teachers’ propensity to use pedagogies 
focussed on real world problem solving and student collaboration. The teachers also 
responded to open-ended questions about how best to engage students in STEM. The 
mathematics teachers also completed an additional survey focussed on their beliefs about 
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the discipline of mathematics and pedagogical approaches. After each on-campus session 
the teachers completed additional surveys asking them to rate the various components of 
the session and to reflect on the impact of the Academy in their schools 


Evaluation Results 


The Academy was attended by 64 teachers from 13 schools (24 mathematics, 23 
science, and 17 technology). Approximately 60% of the teachers were female and the 
mean number of years of teaching experience was 17.9 years (SD = 9.4). The schools 
represented a range of school systems (eight government, four non-government, and one 
Catholic), primarily from the Sydney metropolitan area with only three provincial schools. 

A key aim of the Academy was to increase teachers’ capacity to engage students in 
STEM subjects in school. At the beginning of the first on campus session, teachers (n = 64) 
were asked to respond to the following open-ended question: What are the best ways for 
teachers to promote student engagement? Their responses emphasised the importance of 
focussing on real-life examples that are relevant to students’ lives (m = 20), using hands-on 
activities and ICT where appropriate (n = 13), building a respectful, positive classroom 
environment (n = 9), facilitating an inquiry based approach to learning (n = 7), and student 
collaboration and group work (” = 7). Although the teachers emphasised the value of using 
real-life examples, few of the teachers indicated that they regularly planned for real-world 
connections in the classroom, such as allowing students to consult with experts outside of 
the school setting, involve parents or community members in school activities, listen to 
guest speakers or produce something for use outside of the classroom. Therefore, prior to 
the Academy, despite many teachers acknowledging the value of real-world problem 
solving as a means of engaging students, it seemed that few teachers actively planned for 
real-word interactions in their classrooms. Teachers were more likely to plan for student 
collaboration in the classroom, however, the mathematics teachers were less likely to do so 
in comparison to the science and technology teachers. 

When teachers were asked about the value of the various components of the Academy 
program they consistently responded that they felt that they had extended their knowledge 
of pedagogical approaches for teaching STEM and were excited and enthusiastic about 
trialling their new teaching approaches with their students back in their schools. They were 
less positive about the degree to which their knowledge of STEM content had increased. 
They greatly valued the networking opportunities with other teachers and tended to value 
the collaborative on-campus sessions more highly than plenary/guest speaker sessions. 

I think from talking to the teachers at other schools and getting ideas and sharing ideas and just that 

was just really the best thing. There were things I came away with or things I could help people with 


that you know, it would take a lot of time of your own to be able to achieve. (Science Teacher, Non- 
government school) 


While the teachers were in their schools in between the two on-campus sessions they 
were visited by an Academy mentor and invited to participate in an online community via 
the Edmodo platform. The mathematics (7.6/10) and technology teachers (7.4/10) valued 
their interactions with their mentors more highly than the science teachers (5.6/10), 
perhaps reflecting the degree to which a successful mentor/mentee relationship depends on 
the personnel involved. The teachers appreciated the outsider expertise, a sounding board, 
confirmation, insight and advice, although there were a noteworthy number indicating that 
they didn’t really need a mentor, all from schools fitting a higher SES school category. In 
contrast, there was a substantial number who indicated that they would have appreciated 
more visits or contact with their mentor. Many of these schools were from the lower SES 


spectrum of schools. When interviewed the mentors emphasised the importance of timely 
visits to schools and the importance of school support for teachers as they planned and 
implemented their STEM activities. 
So the fact that you do have mentors is a massive plus in the program. I’m just wondering as we go 
forward, do we actually need more mentors so that perhaps a contact could be a little bit more 
consistent or a little bit more often, and whether or not it's the actual teachers who’ ve been involved 


in this first round —could see how that would be a great advantage if that could happen down the 
track.” (STEM Teacher, Catholic School) 


An Edmodo site was established to facilitate communication and resource sharing 
amongst STEM Academy participants. Forty teachers replied in the survey that they had 
used the site, but of those teachers only 10 replied that they had used the site frequently. 
Thirteen teachers reported that they did not use the Edmodo site at all. Teachers who used 
the site regularly did so because of the great resources being shared. Those that didn’t use 
the site, or who used it infrequently, said in the survey that a lack of time was the main 
reason for not doing so and some experienced problems navigating to the group pages 
within Edmodo. 


That really made me realise how important it is to keep in contact with other maths teachers and the 
Edmodo page that the STEM Academy came up with is great and I’ve actually joined other Edmodo 
pages for maths teachers as a way of — I guess being isolated, that’s one way that I can keep in 
contact with other maths teachers and share ideas and things like that (Mathematics Teacher, 
Government school). 


After the second on campus session the teachers were asked for their reflections on the 
Academy in terms of the impact within their schools. On the positive side, the teachers said 
that there was improved student engagement, more use of interdisciplinary projects 
(although not in mathematics), and increased enjoyment of mathematics, particularly for 
girls, possibly due to the use of more challenging problems. When asked about the 
challenges they encountered, overwhelmingly the teachers cited the lack of time that they 
had for planning, the difficulty in involving other staff in their schools, lack of resourcing 
for new equipment and in some cases, a perceived lack of support from senior staff in their 
schools. 

The students were engaged before, but I think that they are engaged in a slightly different way be- 

cause I’m asking the questions, the students are wanting to find the answers. Instead of just wanting 

to be successful, they want to find an answer, so I think there is a slight difference there in just that 


whole thing. I think that enquiry process is starting to take hold, if you know what I mean (Science 
Teacher, Non-government school). 


Conclusion 


Overall, the first STEM Teacher Enrichment Academy was positively received by the 
participant teachers. However, based on the evaluation conducted — several 
recommendations were made: 

1. to plan for more time in interdisciplinary groups working on real world problem 

tasks; 

2. to include more planning time within the residential schedule so that teachers 

would be more prepared to implement STEM activities in their schools; 

3. to reconsider the use of Edmodo to encourage teachers to use the site more 

frequently and to view the site as a rich source of resources for sharing; 

4. to expand the use of mentors within the Academy to ensure that teachers can access 

this expertise in a timely manner; 


5. to provide teachers with strategies for including and enthusing other staff in their 

schools to use the STEM activities; and 

6. to target disadvantaged and/or isolated schools, where possible, as these teachers 

appeared to gain the most benefit from the Academy. 

On the basis of the recommendations from the evaluation, several changes were 
implemented to better support teachers learning and to enhance the learning experiences of 
students. The Expression of Interest template for the second Academy required schools to 
nominate a STEM Leader who would manage team meetings, encourage teachers to use 
Edmodo for sharing experiences, support teachers to consider using the Academy 
experience for accreditation requirements, and to coordinate the STEM school team’s 
presentation and final report to the Academy. In addition, after schools were selected, each 
school site was visited by a member of the Academy team to meet members of the STEM 
team as well as school Executive members to ensure all were clear about Academy 
requirements before the program began. They were encouraged to pre-plan before coming 
to the first on-campus session by examining each subject area’s scope and sequence, 
identifying common content and processes, comparing assessment requirements, and 
considering when and how they would develop a cross-disciplinary approach to STEM 
teaching and learning. 

During the on-campus session, schools were provided with more time to work in 
school teams to develop cross-disciplinary programs, mentors played an active role in 
working with the school teams and then visiting each school during the two on-campus 
sessions. Some schools from the first Academy were also invited to attend the second 
Academy to share their experiences and provide feedback on early plans. This approach 
extended the network of STEM schools and helped to build the community of practice. 
One of the challenges remaining in the Academy program is to consider ways to sustain 
the early STEM work in each school and to scale the approach to more teachers in each of 
the subject areas. 
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This paper briefly describes the aims and research design for the 2015-2016 STEM Teacher 
Enrichment Academy Evaluation and gives detail about the research design for the case 
study reported herein. A subset of findings from the case study school, developed through 
analysis of student and teacher interviews, is reported to highlight STEM program strengths 
and how the Academy was perceived as contributing to what was achieved. Student 
interviews showed the development or strengthening of STEM students’ career aspirations. 


The underlying intentions of STEM education are to engage students in STEM 
subjects, so they develop deep understandings, and draw flexibly on what they know when 
exploring unfamiliar situations (Freeman, Marginson, & Tytler, 2015). The Melbourne 
Declaration of Educational Goals for Young Australians (Ministerial Council on 
Education, Employment, Training and Youth Affairs, 2008) shares these goals: 
“Successful Learners... are creative, innovative and resourceful, and are able to solve 
problems in ways that draw upon a range of learning areas and disciplines” (p. 8). As 
Anderson (Paper | in this symposium) states, we need to know more about effects of 
interdisciplinary STEM education on outcomes for students. The findings reported herein 
from the STEM Teacher Enrichment Academy Evaluation 2015-2016 (undertaken by 
Tytler and Williams, Deakin University) add to the body of knowledge on how 
interdisciplinary STEM Education might influence the potential for STEM career 
aspirations of students. 


Foci of the Evaluation 


The second STEM Academy Evaluation was designed to find out more about how the 
STEM Academy experience: (a) led to changes in pedagogical practices, (b) supported the 
design and implementation of interdisciplinary projects, (c) encouraged collaboration 
between teachers in STEM disciplines within and beyond the school, and (d) influenced 
student engagement with STEM subjects. This paper focuses within that evaluation on: 
“What influences does the model of interdisciplinary STEM education, as developed and 
implemented in the case study reported, have on student engagement in STEM subjects?” 


Research Design Elements 


Design of the evaluation. Data for the broader study included document analyses, 
survey and questionnaire responses, field notes including photos, and interviews with 
various stakeholders. Both the Component Mapping teacher survey and the Change in 
Teacher Practices Questionnaire (administered after each of the two workshops) informed 
probing in interviews and foci of attention for STEM class observation. The survey (Tytler 
et al., 2004) and the questionnaire (Williams, 2000) were refined from research tools 
previously developed by evaluation team members. The survey captures where teachers 
perceive they lay along various evidence based spectra of high quality pedagogical 
practices. The Change in Teacher Practices Questionnaire was adapted from a 
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questionnaire on student learning change during problem solving employed previously by 
Williams (e.g., Williams, 2000)). Teachers were asked to identify what they had learnt 
from Academy participation, and what had influenced that learning. This questionnaire and 
interviews with teachers and students provided opportunities for them to identify what they 
attended to in STEM Ed in the school rather than only respond to researcher identified 
elements. 

Case study design. The case study design as implemented included observations of 
STEM Ed activity (in STEM-Tech and STEM-Maths classes), and interviews with various 
school community members: a) school leaders directly involved in supporting STEM 
initiatives; b) Academy teachers (which included school leaders); and c) six students from 
STEM classes (three boys, three girls) who were identified by the teachers as 
demonstrating engagement with STEM subjects. Engagement was also identified through 
teacher and student interviews, and observed body language in interviews and classroom 
observations through focus of attention, body direction, lack of awareness of activity 
external to their focus, connecting to the ideas of others, and exclamations (EyYDUPLEx 
Framework: Williams, 2003). The interviews included probes about what had influenced 
new learning that occurred. This helped identify links between engagement and creative 
STEM activity. 


STEM-ED 


The school’s Year 7/8 STEM program (STEM Ed) commenced two years prior to 
Academy participation. It was funded by the supportive principal who had faith in the 
primary STEM-ED instigator (now Vice Principal: VP), who developed the program in 
conjunction with the heads of technology and science. The team perceived benefits from 
joining the Academy: feelings of increased obligation (thus motivation) to succeed because 
they were given this opportunity, increased confidence that the innovation was worthwhile 
because the Academy selected them, access to a network of schools with similar interests 
and challenges through the Academy, raised STEM education profile at school (which 
encouraged more teachers to commit to STEM), valuable PD for staff, and additional time 
for team planning. 

Given timetable and resource constraints in this small school, STEM-ED was located 
in the classes of each discipline, renamed STEM-Tech, STEM-Science, and STEM-Maths. 
Activity undertaken in these classes was interdisciplinary. For example, during data 
collection for the case study, the Year 7/8 STEM-Ed group were undertaking a multi- 
disciplinary project on Space, and Rockets. Each of STEM-Tech, STEM-Science, and 
STEM-Maths focused within this topic in ways that interconnected the STEM subjects. 

STEM-Tech groups of three or four built rockets to launch to test their capacity. The 
teacher posed questions rather than gave answers when students struggled to find ways to 
proceed. Student 1’s (S1’s) eyes lit up and his voice became animated (demonstrating 
engagement with STEM-Ed) as he described this program: “It’s more open [than other 
subjects] ... you got to think out ways to do it for yourself rather than be taught a certain 
way by the teacher”. A rocket design sketch (with design features justified), and rocket 
construction are shown (Figure 1, left and right, respectively). Learning included: “centre 
of mass and ... thrust need to align so rocket does not tip ... wings need to be down low as 
stabilisers” (S1). 

STEM-Science students tested materials to help make decisions about what materials 
to use for their rocket: “We were testing how fire-resistant leather was — put leather over a 
Bunsen burner and it shrivelled up still intact” [S3]. They also studied the Solar System. 


Figure I. STEM-Tech design of one student group and rocket assembly of another group. 


STEM-Maths students in groups of four, allocated group roles. They made a calendar 
for Mars using conversions from Earth to Mars time. The teacher stimulated class 
discussion by raising questions from these discussions, for students to discuss and resolve. 

The number of months were completely up for grabs for a start- we talked about that and well okay 

Mars has how many moons ... [and] does Mars have seasons and what are seasons and do they occur 

on every planet and we found that Mars does have solstices and equinoxes but they do not quarter 


the year as ours do ... it was a class conversation I ran and there were times where I spotted the 
questions to ask and they discussed and ran with that. [Teacher of Class, Vice Principal] 


Group roles included a “think big” group member who selected a focus to explore beyond 
the core requirements of the project. S1 decided to explore Mars leap years. 


Design Thinking Embedded in STEM-ED 


All six students drew attention to autonomy enabled in STEM-Ed and the focus on 
learning for a purpose rather than just learning for assessment. Learning in STEM-ED was 
described in various ways: “Compared to learning in other classrooms this [STEM-ED] 
tries to use your brain more, it challenges you more” [S2]. ‘Using what is learnt rather than 
just learning it’ was a common comment made: 

I just like the whole STEM program because you don’t just get to learn stuff in the classroom and 


not do anything with it- in STEM you learn information but then you get to put it into a practical use 
in Tech [S6] 


The STEM-ED team devoted time and energy to familiarising other teachers with 
Design Thinking. Student comments showed Design Thinking was embedded in STEM- 
ED. They either made explicit references to it or described the process in activity reported: 

liked the immersion ... how you do things ... the design where you get told everything then do two 


designs on paper if possible a scale and then pick one of them and that usually takes most of the 
time ... then testing and readjusting (S4: female student) 


S6 captures the cyclical nature of the process: trying to solve a problem within certain 
constraints, testing products to work out how to proceed in environmentally friendly ways: 
we are given like strict materials that we can use ... it helps us to think about ‘how to use this in a 
productive way like an effective way so that we don’t waste any materials but ... if it doesn’t work 


we try not to use more materials but if we have to use more materials then we will but we are trying 
to make it as effective as we can 


The design process appeared crucial to changed aspirations of students not previously 
considering STEM careers. 


STEM Career Aspirations 


This different way of learning shifted the career aspirations of the three girls. The boys 
and one girl (S6) were already interested in at least one STEM discipline at the start of 
secondary school. S6 was interested in Science, but her greater interest was English at the 
start of secondary school. The boys displayed their interest through their voices becoming 
more intense and their faces more animated as they discussed various STEM-ED activities. 
STEM-ED changed the career aspirations of all students interviewed with each of them 
shifting more towards pursuing STEM careers, For example, S4: 


“(In primary school, I preferred] Art [as] more creative but ... now with science not so right and 
wrong ... more like creativity ... I like Art still ... but Sci and Tech I like ... now and careers in that 
area.” 


S6 who wanted to be a teacher or missionary nurse included engineering as an aspiration 
after the excursion to the university drew her attention to engineers helping others: 


I wasn’t really thinking about being an engineer in primary school but then when we had the STEM- 
ED camp ... visit{ed] the university and we got to see ... different engineers and what projects they 
do ... it really like inspired me because it ... showed me that we don’t just learn this stuff in school- 
and then not use it in real life- you can ... use the information you have learnt ... to make a difference 


Concluding Comments 


This study shows that this interdisciplinary STEM education model that physically 
located classes in each discipline, while employing interdisciplinary projects that 
emphasised design thinking, achieved student outcomes consistent with the broader STEM 
agenda. The processes employed increased/strengthened STEM career aspirations for both 
boys and for girls. Creative opportunities involving design thinking (S4, S5) and raised 
awareness that engineering can involve helping others (S6) influenced these changes. The 
University of Sydney STEM Teacher Enrichment Academy contributed to team 
opportunities to achieve these changes by affirming the directions the team were taking, 
raising the STEM school profile, and resourcing the project. Preliminary outcomes for 
students interviewed indicate that selection by the Academy and the way teachers have 
used this opportunity has raised student awareness of STEM and stimulated interest in it. 
The team look forward to finding which senior secondary subjects STEM-ED students will 
select. 
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Developing an Evaluation Framework for Future STEM Academies 
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Building upon prior evaluations, a comprehensive evaluation plan for the STEM Teacher 
Enrichment Academy at the University of Sydney is currently being devised and will 
consider the multiple perspectives of teachers, students and school leaders, and the 
interplay between these key stakeholders as it affects program outcomes. Additionally, 
effects of parents and industry partners will also be examined. The evaluation will follow a 
mixed-methods protocol with the additional collection of pertinent school-level data of all 
participating schools. Evaluation results will not only prove beneficial in shaping future 
academies but will also add to the literature in this growing field of academic research. 


The evaluations conducted as part of the first and second STEM Teacher Enrichment 
Academies offered key formative assessment data that proved useful in improving the 
structure and content of the academy as seen through the lens of the academy participants. 
As we move forward in developing an evaluation framework for future academies, our 
focus will become more outward with the aim of evaluating not only the initial objectives 
of the academy but also keys goals as outlined by the National STEM School Education 
Strategy (National Council, 2015). 

The professional development offered through the STEM Teacher Enrichment 
Academy endeavours to increase teachers’ pedagogical content expertise through guiding 
teams of teachers in their development and delivery of integrated STEM units of study 
within each of their schools. While it is anticipated that this approach would inspire 
teachers to expand their own personal interests in STEM, for some teachers this team- 
styled approach towards creating and disseminating integrated STEM content in their 
classrooms may be a novel experience affecting their personal beliefs towards teaching, as 
well as their understanding and knowledge towards their subject area. The literature is 
replete with examples of how teacher beliefs and self-efficacy significantly impact student 
learning and achievement, and we assume likewise in our evaluation (Tschannen-Moran & 
Barr, 2004). 

As the need for an expanded STEM workforce grows, the role that teachers play 
towards encouraging students to pursue STEM fields of study appears critical. In building 
an evaluation model, emphasis will be given not only to teachers’ perceptions of their own 
capabilities and beliefs both individually and collectively, but also towards their capacity 
in affecting students’ interests, motivation, 21st century learning and future career choices 
in STEM (Betz & Hackett, 1986; Dick & Rallis, 1991; DuFour & DuFour, 2010). 

One of the major outcomes envisioned of the STEM Teacher Enrichment Academy is 
an eventual interest and increase in student participation in STEM occupations. 
Accordingly, an assumed pathway towards that goal is increased student engagement with 
senior school mathematics, science and technology subjects. The attitudes students possess 
towards STEM are a significant factor in not only influencing future STEM subject choice 
but also in students’ pursuit of STEM related careers (Maltese & Tai, 2011), and as such 
become an important area for examination in our evaluation plan. 

Additionally, school principals are appropriately considered as drivers who may 
influence the effectiveness of professional development programs aimed at student success 
and teacher growth in STEM education (Prinseley & Johnston, 2015). Without the 
advocacy, vision and leadership of principals in STEM curricular efforts, the impact of the 


(2017). In A. Downton, S. Livy, & J. Hall (Eds.), 40 years on: We are still learning! Proceedings of the 40th 
Annual Conference of the Mathematics Education Research Group of Australasia (p. 599-602). Melbourne: 
MERGA. 


STEM Teacher Enrichment Academy’s professional development program in any one 
school may be short-lived. The academy aims to provide teachers with the tools, resources 
and knowledge to affect positive change in their classroom teaching through encouraging 
collaborative hands-on STEM learning in a “real-world” context. Yet, the leadership, 
condition and culture within each particular school context may also influence the eventual 
outcomes of this STEM focused professional development program. Therefore, when 
evaluating the program effectiveness of the STEM Teacher Enrichment Academy it seems 
essential to consider and understand the multiple perspectives of teachers, students and 
school leaders, and the interplay between these key stakeholders as it affects program 
outcomes. 


Designing an Evaluation Framework for the 
STEM Teacher Enrichment Academy 


Integrated STEM teaching and learning is a new endeavour for many Australian 
schools, and as such their effects are under-researched. While the quality and scope of 
STEM education evaluation schemas varies widely (Brody, 2006), the conclusions drawn 
by Bryk and his fellow researchers (2010) offer a concise reflection on the multiple factors 
considered as pivotal in successful science and mathematics educational programs: (a) 
school leadership as an impetus for change, (b) professional capacity of faculty through 
engagement with professional development, changes in values and beliefs, and the ability 
for collaboration amongst faculty, (c) outreach that strengthens the ties between parents, 
community, academic institutions and industry, (d) student-centred learning environments, 
and (e) instructional guidance that advances learning. In addition to these attributes, two 
specific STEM evaluation models with both theoretical and systems-based approaches, 
offer beneficial insight as we devise our own evaluative strategies (Arshvansky et al., 
2014; Saxton et al., 2013). 

Our specific evaluation plan for the STEM Teacher Enrichment Academy is designed 
as a mixed-methods protocol, with both survey and interview components, and will 
measure outcomes for principals/school leaders, teachers and students based on key 
program objectives. Additionally, effects of parents and industry partners will also be 
examined (see Figure 1). 
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Figure 1. STEM Teacher Enrichment Academy comprehensive evaluation model. 


Principals/School Leaders. The survey items designed to measure the advocacy, vision 
and leadership of school principals are derived from the P-STEM survey instrument which 
specifically measures the leadership offered by school principals towards STEM education 
within their specific school contexts (Friday Institute for Educational Innovation, 2014). 
Using a 5-point Likert scale, principals indicate their response to items such as, “Regarding 
the STEM work at my school, I... enable collaboration of teachers across content areas.... 
ensure technical support/other resources are available for STEM teaching...maintain 
strategic partnerships with STEM industries”. Reliability testing has produced a 
Cronbach’s alpha above 0.90 for this instrument. The 37-item survey will be administered 
to principals prior to their school’s enrolment in the Academy and one year after their 
school’s participation, and will also serve as a formative reflection for principals. 
Additionally, principals will be interviewed to capture a more nuanced understanding of 
the specific adaptive STEM culture within their schools, and their partnerships with STEM 
specific industries. 

Teachers. The teacher questionnaire will consist of items from the T-STEM survey 
instrument (Friday Institute for Educational Innovation, 2012b) and the Collective Teacher 
Efficacy Measure (Goddard, Hoy, & Hoy, 2000). The T-STEM survey assesses teachers’ 
personal STEM teaching efficacy beliefs, STEM teaching outcome expectancy beliefs, 
reflection on STEM instruction, 21st century learning attitudes, teacher leadership attitudes 
and STEM career awareness. Reliability testing on each of these scales has produced 
Cronbach’s alphas ranging from 0.814 to 0.948. The Collective Teacher Efficacy Measure 
is comprised of four subscales that assess group competence and task analysis. The survey 
will also contain four reflective open-ended prompts. The teacher questionnaire will be 
administered during the on-campus components of the academy and one year post- 
academy participation. Additionally, teachers will be interviewed at the conclusion of their 
school’s involvement with the academy to probe the STEM initiatives in their school, their 
perceptions of the impact of the STEM Teacher Enrichment Academy, adjustments made 
to their school’s curriculum to accommodate STEM teaching and learning, their 
involvement with communities of practice, partnerships with industry, and efforts to 
sustain STEM initiatives in their schools. 

Students. A student questionnaire will be devised of items from the S-STEM survey 
(Friday Institute for Educational Innovation, 2012a) and the STEM Semantics survey 
(Tyler-Wood, Knezek, & Christensen, 2010). The S-STEM survey measures student 
attitudes in mathematics, science and technology, future career interest, self-assessment of 
current achievement across STEM subjects, future projected STEM subject uptake and 
21st century skills. CFA Goodness of Fit Indices and Cronbach’s alphas indicate a high 
level of validity and reliability of this instrument. The STEM Semantics survey assesses 
student perceptions and attitudes towards each of the separate STEM disciplines in which 
students indicate on a scale of | to 7, for example, their opinion of science as fascinating 
(1) to mundane (7). Surveys will be administered to students prior to and after their 
engagement with integrated STEM teaching and learning. 

Parents. Parents are certainly influential in their children’s career selection, 
particularly for students choosing an engineering or science pathway (Dick & Rallis, 
1991). The STEM-CAT survey (White, 2015) will assess parents’ beliefs about STEM 
education, their values towards STEM education and their perception of the resources that 
their child’s school offers in STEM education. 

School Level Data. As the numbers of schools and educators who participate in the 
STEM Teacher Enrichment Academy grows, so does the need to create a database in order 


to track the long-term impact of student and teacher engagement with integrated STEM 
teaching and learning. Data to be collected will include school level data such as 
demographics of student body, school affiliation, offerings and enrolments in STEM 
subjects, and faculty level data. 

The data collected as part of the STEM Teacher Enrichment Academy evaluation plan 
will allow us to assess both short and long-term effects of the academy. Principal, teacher 
and student surveys will be administered both pre-test and post-test allowing for 
comparison through repeated measures statistical testing. Further inferential testing may 
reveal connections across the participant data. Interviews will further elucidate quantitative 
findings. The data gathered through the STEM Teacher Enrichment Academy evaluation 
will not only prove beneficial in shaping future academies but will also add to the literature 
in this growing field of academic research. 
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